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Abstract 


The efficacy of objective structured skill examination (OSSE) was evaluated as a new and useful exami- 
nation. The second grade students at Kinki University Faculty of Medicine were prospectively eligible. 
Students took problem-based learning (PBL) tutorials in addition to classroom lectures and laboratory 
practice in each session. The sessions were on: cardiac, circulation, hematological, and renal systems. At 
the end of each session, students took the paper examination as a part of conventional evaluation. After 
all sessions, OSSE was performed to evaluate the students’ performance. The technical skills evaluated 
in OSSE were as follows: electrocardiogram recording, blood pressure measuring, use of centrifuge, 
pipette technique, hematocrit measurement, urinary specific gravity determination, and measurement of 
urine volume. One order sheet and the experimental apparatus for OSSE were positioned at each sta- 
tion and students terminate the maneuver in 2 min. The score from OSSE was compared to that from the 
conventional evaluation. Significant positive correlation was not observed. More than half of the students 
considered OSSE to be meaningful. In conclusion, we developed a new structured examination to objec- 
tively assess the students’ technical skills in laboratory practice. 

Key words: curriculum reform, laboratory practice, objective structured skill examination, physiology. 


Introduction 
Recent medical education in Japan has shown a dramatic change reflecting various fac- 
tors including medical malpractice (Watts 2003). The most distinctive aspect of the new curricu- 


lum is the introduction of problem-based learning (PBL) tutorial in medical education (Matsuo 
2002). PBL was introduced in the medical education of Kinki University School of Medicine in 
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1998 (Matsuo 2000). In our PBL for physiology, medical students take conventional classroom 
lectures and accompanying laboratory practices (often referred to as hybrid PBL). In 2001, the 
Coordinating Council on Medical and Dental Education designed the model core curriculum, 
which is highly recommended to be the central focus of the curriculum of each medical school 
in Japan. Since then, most medical schools in Japan have undertaken restructuring of their cur- 
riculum from the traditional style to the newly proposed innovative style (Matsuo et al. 2000; 
Onishi et al. 2004). The most highly recommended part in our new curriculum is the introduc- 
tion of PBL in teaching medicine. The main subject of our newly reformed curriculum in Kinki 
University School of Medicine is to achieve the integration of better knowledge in medicine. 
Compared to the traditional curriculum for medical education, there was significant superiority 
in both short-term and long-term memory for the student who studied via our new PBL oriented 
curriculum (Fukuda et al. 2002). 

At the second year of medical school, organ-oriented lectures were presented to the stu- 
dents (Kozu 2006; Teo 2007), and they could learn not only the physiological or biochemical 
function but also the anatomical or histological structure for each organ in the human body in 
the series of classes within the week. In the former educational style, students learned medicine 
based on the subjects they took, e.g., biochemistry, physiology, and anatomy. Within the ses- 
sion, we coordinated the relevance of all study subjects so that students could resolve their own 
practical queries in medicine on their own. 

Compared to textbook-based or lecture-based learning, learning physiology through ob- 
taining laboratory techniques is more practical and active. We consider supplemental skill train- 
ing hours (from here on designated as “laboratory practice”) to be very important to understand 
medical subjects including physiology. It is because students can experience many medically- 
related skills in laboratory practice. However, assessing the student’s laboratory practice skills 
is actually difficult due to unequal participation. To evaluate the performance of students in the 
laboratory practice, and to evaluate individual contribution objectively, we have developed an 
objective structured skill examination (OSSE). This is performed in a small room where experi- 
mental apparatus are placed on the table with an order sheet, and 2 min is allowed to perform 
without any oral indication. 


Methodology of Research 


After obtaining approval from the education committee at Kinki University School of 
Medicine, the second grade medical students were prospectively eligible. Students took four 
sessions in the structure and functional unit, including cardiac system, circulation system, he- 
matological system and renal system. Each session included each of PBL, conventional class- 
room lecture and laboratory practice. At the end of each session, students took the paper exami- 
nation. Evaluation of the PBL was based on: attendance, performance, and understanding of the 
tutorial weighing 40%, 30% and 30%, respectively. PBL directors assessed the students during 
PBL. Evaluation of the laboratory practice was based on: attendance, attitude towards the prac- 
tice and content of report weighing 30%, 30% and 40%, respectively. Laboratory instructors 
evaluated the students after completion of the laboratory practice. The weight proportion of 
each item in the scoring system was 60% for paper examination, 30% for PBL and 10% for 
laboratory practice. The above-mentioned is our ordinary assessment system for the physiology 
class (designated as “conventional evaluation’’), and the performance of OSSE of the students 
was statistically compared to the score of the conventional evaluation for each session. 

The new assessment system, OSSE, was used to evaluate the student’s performance. 
It was conducted after all sessions of the structure and functional unit were completed. Six 
structured stations were prepared for OSSE: electrocardiogram recording as cardiac system 
evaluation, blood pressure measuring as circulation system evaluation, use of centrifuge, pi- 
pette technique and hematocrit measurement as hematological system evaluation and urinary 
specific gravity determination as evaluation of the renal system. There were five checkpoints at 
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each OSSE station. The following is an example of the order sheet, which was for the electro- 
cardiogram recording: “When you record the three bipolar limb leads of the electrocardiogram, 
such as lead I, attach the negative, or ground electrode at the proper body position. When you 
record V2, V4, and V6 leads, which are unipolar chest leads, place each electrode at the proper 
body position.” 

Guidance for each laboratory technique was given during the laboratory practice in each 
session. One task order with the experimental apparatus for the OSSE was placed at each sta- 
tion. The checkpoints were not announced to the students. A time period of 2 min was allowed 
for each student in each task. Exact scoring of the OSSE was based on the student’s task per- 
formance, and the examiner evaluated the students with five structured checkpoints at each 
workstation. A maximum of two points was given for each checkpoint (good skill for 2, fair for 
1, and poor for 0), thus the score varied from 0 to 10 for each station. The correlation coefficient 
between score from OSSE and that from conventional estimation was statistically analyzed for 
each session using Pearson correlation test. Furthermore, the correlation coefficient between 
score from OSSE and that from laboratory practice was statistically analyzed for each session 
using Pearson correlation test. A p-value of less than 0.05 was regarded as statistically signifi- 
cant. All of the examinations for OSSE were completed in one day for each subject in order to 
prevent study bias. Students were prohibited to have contact with each other on the examination 
day. To achieve this, students were allowed to move in only one direction without contacting 
other students and cell-phone was not brought to OSSE. 

In order to assess the quality of OSSE, students were requested to complete a question- 
naire (Table 1) at the end of OSSE. 


Table 1. Questionnaire for Objective Structured Skill Examination (OSSE). 





Q1. How tense were you during OSSE? 
a. Highly tensed b. Tensed c. Moderate d. Relaxed e. Very relaxed 


Q2. How did you participate in the laboratory practice? 
a. Actively participated b. Participated _c. No interest either way 
d. No participation —_e. Absolutely no participation 


Q3. If you had been informed of the OSSE checkpoints prior to the laboratory practice, how would you have 
participated? 

a. Actively participated b. Participated _c. No interest either way 

d. Probably would not have participated _e. Definitely would not have participated 
Q4. How difficult was OSSE? 

a. Very easy b. Easy c.Noteasy -not difficult d. Difficult e. Very difficult 

Q5. What did you think about the length of OSSE? 

a. Very short b. Short c.Notshort-notlong d.Long e. Very long 

Q6. What did you think about the contents of OSSE? 

a. Very good b. Adequate c. Neither adequate nor inadequate d. Inadequate 

e. Very inadequate 

Q7. How long did you spend reviewing for OSSE? 

a. More than 3 hours b.2-3hours c.1-2hours d. Less than 1 hour 

e. Did not review 

Q8. Do you think that the examiner can evaluate your practical skill by OSSE? 
a. Yes b.| think so. c. No idea. d. | don’t think so. 

e. No, not at all. 

Q9. How will you participate in the practice in the future? 

a. Actively participate b. Participate c. No interest either way 

d. Probably will not participate e. Definitely will not participate 

Q10. What did you think about quantitative evaluation for your laboratory skill? 
a. Very good b.Good c.Notgood-norbad d.Bad e. Very bad 
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The questionnaires included individual motivation toward the laboratory practice (Q2), 
difficulty of OSSE (Q4), validity of OSSE in evaluating personal skills (Q8 and Q10), time 
spent reviewing laboratory practice for OSSE (Q7), and attitude towards future laboratory prac- 
tice (Q9). The answers included the following choices: a. very good, b. good, c. fair, d. poor, e. 
very poor. 

Results of Research 
The distribution of each score from OSSE and from traditional estimation in renal course 


is Shown in Figure 1, which shows no significant correlation between them. The pattern of the 
relation is rather common though a slope of correlation line is different. 
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Figure 1. Distribution of each score from OSSE and from conventional estima- 
tion in renal course. 


The full score from OSSE and conventional estimation was 100 and 1,000 points, re- 
spectively. The score of conventional estimation represents the sum of the scores from written 
examination, PBL tutorial and laboratory practice. 

The correlation coefficients between the results of OSSE and those of conventional 
scores in each session are shown in Table 2. The correlation coefficient ranged from 0.028 
(p=0.789) for the renal session to 0.134 (p=0.195) for the hematology session. 


Table 2. Correlation analyses between scores from OSSE and scores from con- 
ventional estimation. 








Cardiology course Circulatory course Hematology course Renal course 
r 0.113 0.120 0.134 0.028 
p 0.272 0.246 0.195 0.789 
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The correlation coefficients between the results of OSSE and those of laboratory practice 
in each session are shown in Table 3. The correlation coefficients ranged from 0.005 (p=0.959) 
for the renal session to 0.232 (p<0.05) for the circulatory session. In one-fourth of the sessions, 
the scores from OSSE were significantly (p<0.05) positively correlated with those from labora- 
tory practice. In another session, no significant correlation was observed. 


Table 3. Correlation analyses between scores from OSSE and scores from labo- 
ratory practice. 








Cardiology course Circulatory course Hematology course Renal course 
r 0.181 0.232 0.154 0.005 
p 0.077 0.022* 0.143 0.959 
* p<0.05 


According to the results of the questionnaire, about 60% of the students had a positive 
attitude (actively participated and participated) towards the laboratory practice. More than half 
of the students answered that the quantitative evaluation for their practical skills was very good 
or good (Figure 2). About 20% of students had spent more than two hours reviewing for OSSE 
beforehand. Only about 10% of students thought that OSSE was easy. About 90% of the stu- 
dents indicated that they would positively participate in the future practice. 
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Figure 2. Responses to question 10, “What did you think about quantitative 
evaluation for your laboratory skill?” 


Five choices for this question are shown in the axis of abscissa. The selection percentage 
for each choice was calculated. 
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Discussion 


It is strongly thought that laboratory practice is as valuable as PBL. The objective struc- 
tured clinical examination is widely adopted in medical education in the United States (Battles 
2004) and it is believed that obtaining practical skills for communicating with patients is as 
important as obtaining medical knowledge (Barzansky et al. 2003; Barzansky et al. 2009). 
Likewise, acquiring practical techniques in the laboratory is extremely beneficial for the medi- 
cal students (Harden 1975). It is clear that students can learn a broad spectrum of techniques 
during the laboratory practice. However, it is also obvious that there are variations in the level 
of gained techniques among the students. Therefore, our investigation for the OSSE is mean- 
ingful from the viewpoint of a pilot study to evaluate students’s skill objectively. 

OSSE evaluated the laboratory skills of the students. No significant correlations were 
observed between scores of OSSE and those from conventional estimation (Table 1). Further- 
more, only one-fourth of session showed significant correlation between scores of OSSE and 
those from laboratory practice (Table 2). These findings indicate that students who obtained 
high score in conventional estimation or laboratory practice may not necessarily have good 
laboratory skills. 

Not only knowledge but also technical skill is necessary to achieve a high score in OSSE. 
Since medical doctors need practical skills, and students would try hard to acquire knowledge 
and skills tested in exams, OSSE may be very important for the acquisition of practical skills. 
Figure 5 indicates that most of the students actually did spend time to review because of OSSE. 
However, due to the small number of collected data and selected tasks, further investigations 
are obligatory to conclude the efficacy of OSSE. Hopefully, this pilot study will prompt other 
medical schools to follow suit in the future. 

Significant negative correlations were not observed in any of the sessions (Table 1). This 
observation suggests that OSSE does not lead to incorrect judgments. Namely, the low level 
students in the conventional estimation were not evaluated as having high skills in OSSE. 

Maintaining a high motivation for laboratory practice can be difficult for medical stu- 
dents. Unfortunately, due to limited resources in space and equipment, some students actively 
participate in the laboratory practice, while others in the same group have a passive attitude, and 
merely observe. Thus, the other purpose of introducing OSSE in the medical education in Kinki 
University School of Medicine was for medical students with high motivation and achievement 
to be objectively recognized by the OSSE score. In fact, as a result according to the question- 
naire shown, the percentage of students who will actively participate in future practice (90%) 
was greater than those who did participate in the present unit practice (~70%). This finding 
suggests that the majority of test takers considered OSSE as a motivation and actively partici- 
pated in the laboratory practice. In addition to our primary purpose of the OSSE, which is to 
objectively evaluate the student’s laboratory techniques, this secondary effect of OSSE seems 
to also be beneficial for medical education. 

Since more than half of the students answered that structured evaluation for practical 
skills was very good or good (Figure 2, Q10 a,b), they considered OSSE as a meaningful 
evaluation. This indicates that OSSE is a good tool to evaluate the practical skill of students in 
physiology education. 

In summary, a new method of examination, OSSE was developed to objectively assess 
the students’ technical skills in the laboratory practice in physiological education. OSSE is pref- 
erably supported by medical students and is thought to enhance the student’s motivation level. 
Furthermore, the present study suggests that OSSE can evaluate the practical skills of students 
that cannot be assessed by the conventional estimation. 
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Conclusion 


In medical physiology education, laboratory practice is very important to understand the 
medical subjects including various function of human. However, it is very difficult to evaluate 
the performance of each students in the laboratory practice, and to evaluate individual con- 
tribution objectively. To dissolve this issue as described above, an Objective Structured Skill 
Examination (OSSE) has been developed. OSSE can evaluate the practical skills of students 
that cannot be assessed by the conventional estimation. 
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